This border is used to show that the element is focused (ie you can type in the input or press the button with Enter). You can remove it with outline property.
What does the outline property do? It provides visual feedback for links that have focus when navigating a web document using the TAB key (or equivalent).
none - Defines no outline; hidden - Defines a hidden outline. The following example shows the different outline-style values: Example. Demonstration ...